Estimating Stress in Online Meetings by Remote Physiological Signal and Behavioral Features

Work stress impacts people's daily lives. Their well-being can be improved if the stress is monitored and addressed in time. Attaching physiological sensors are used for such stress monitoring and analysis. Such approach is feasible only when the person is physically presented. Due to the transfer of the life from offline to online, caused by the COVID-19 pandemic, remote stress measurement is of high importance. This study investigated the feasibility of estimating participants' stress levels based on remote physiological signal features (rPPG) and behavioral features (facial expression and motion) obtained from facial videos recorded during online video meetings. Remote physiological signal features provided higher accuracy of stress estimation (78.75%) as compared to those based on motion (70.00%) and facial expression (73.75%) features. Moreover, the fusion of behavioral and remote physiological signal features increased the accuracy of stress estimation up to 82.50%. © 2022 Owner/Author.

Camera-Based Blood Pressure Estimation via Windkessel Model and Waveform Features

Blood pressure (BP) is generally regarded as the vital sign most strongly correlated with human health. However, for decades, BP measurement has involved a cuff, which causes discomfort and even carries a risk of infection, given the current prevalence of COVID-19. Some studies address these problems using remote photoplethysmography (rPPG), which has shown great success in heart rate detection. Nevertheless, these approaches are not robust, and few have been evaluated with a sufficiently large dataset. We propose an rPPG-based BP estimation algorithm that predicts BP by leveraging the Windkessel model and hand-crafted waveform characteristics. A waveform processing procedure is presented for the rPPG signals to obtain a robust waveform template and thus extract BP-related features. Redundant and unstable features are eliminated via Monte Carlo simulation and according to their relationship with latent parameters (LSs) in the Windkessel model. For a comprehensive evaluation, the Chiao Tung BP (CTBP) dataset was constructed. The experiment was conducted over a four-week period of time to evaluate the validity period of the personalization in our system. On all the data, the proposed method outperforms the benchmark algorithms and yields mean absolute errors (MAEs) of 6.48 and 5.06 mmHg for systolic BP (SBP) and diastolic BP (DBP), respectively. The performance achieves a 'B' grade according to the validation protocol from the British Hypertension Society (BHS) for both SBP and DBP. © 1963-2012 IEEE.

Cost-Effective Solution of Remote Photoplethysmography Capable of Real-Time, Multi-Subject Monitoring with Social Distancing.

Recent advances in remote-photoplethysmography (rPPG) have enabled the measurement of heart rate (HR), oxygen saturation (SpO2), and blood pressure (BP) in a fully contactless manner. These techniques are increasingly applied clinically given a desire to minimize exposure to individuals with infectious symptoms. However, accurate rPPG estimation often leads to heavy loading in computation that either limits its real-time capacity or results in a costly setup. Additionally, acquiring rPPG while maintaining protective distance would require high resolution cameras to ensure adequate pixels coverage for the region of interest, increasing computational burden. Here, we propose a cost-effective platform capable of the real-time, continuous, multi-subject monitoring while maintaining social distancing. The platform is composed of a centralized computing unit and multiple low-cost wireless cameras. We demonstrate that the central computing unit is able to simultaneously handle continuous rPPG monitoring of five subjects with social distancing without compromising the frame rate and rPPG accuracy.

Camera-based Blood Pressure Estimation via Windkessel Model and Waveform Features

Blood pressure (BP) is generally regarded as the vital sign most strongly correlated with human health. However, for decades, BP measurement has involved a cuff, which causes discomfort and even carries a risk of infection, given the current prevalence of COVID-19. Some studies address these problems using remote photoplethysmography (rPPG), which has shown great success in heart rate detection. Nevertheless, these approaches are not robust, and few have been evaluated with a sufficiently large dataset. We propose an rPPG-based BP estimation algorithm that predicts BP by leveraging the Windkessel model and hand-crafted waveform characteristics. A waveform processing procedure is presented for the rPPG signals to obtain a robust waveform template and thus extract BP-related features. Redundant and unstable features are eliminated via Monte Carlo simulation and according to their relationship with latent parameters in the Windkessel model. For a comprehensive evaluation, the Chiao Tung Blood Pressure (CTBP) dataset was constructed. The experiment was conducted over a four week period of time to evaluate the validity period of the personalization in our system. On all the data, the proposed method outperforms the benchmark algorithms and yields mean absolute errors of 6.48 mmHg and 5.06 mmHg for SBP and DBP, respectively. The performance achieves a “B”grade according to the validation protocol from the British Hypertension Society for both SBP and DBP. IEEE

ReViSe: Remote Vital Signs Measurement Using Smartphone Camera

Remote Photoplethysmography (rPPG) is a fast, effective, inexpensive and convenient method for collecting biometric data as it enables vital signs estimation using face videos. Remote contactless medical service provisioning has proven to be a dire necessity during the COVID-19 pandemic. We propose an end-to-end framework to measure people's vital signs including Heart Rate (HR), Heart Rate Variability (HRV), Oxygen Saturation (SpO2) and Blood Pressure (BP) based on the rPPG methodology from the video of a user's face captured with a smartphone camera. We extract face landmarks with a deep learning-based neural network model in real-time. Multiple face patches also called Regions-of-Interest (RoIs) are extracted by using the predicted face landmarks. Several filters are applied to reduce the noise from the RoIs in the extracted cardiac signals called Blood Volume Pulse (BVP) signal. The measurements of HR, HRV and SpO2 are validated on two public rPPG datasets namely the TokyoTech rPPG and the Pulse Rate Detection (PURE) datasets, on which our models achieved the following Mean Absolute Errors (MAE): a) for HR, 1.73Beats-Per-Minute (bpm) and 3.95bpm respectively;b) for HRV, 18.55ms and 25.03ms respectively, and c) for SpO2, an MAE of 1.64% on the PURE dataset. We validated our end-to-end rPPG framework, ReViSe, in daily living environment, and thereby created the Video-HR dataset. Our HR estimation model achieved an MAE of 2.49bpm on this dataset. Since no publicly available rPPG datasets existed for BP measurement with face videos, we used a dataset with signals from fingertip sensor to train our deep learning-based BP estimation model and also created our own video dataset, Video-BP. On our Video-BP dataset, our BP estimation model achieved an MAE of 6.7mmHg for Systolic Blood Pressure (SBP), and an MAE of 9.6mmHg for Diastolic Blood Pressure (DBP). ReViSe framework has been validated on datasets with videos recorded in daily living environment as opposed to less noisy laboratory environment as reported by most state-of-the-art techniques.

Evaluation of a Camera-Based Monitoring Solution Against Regulated Medical Devices to Measure Heart Rate, Respiratory Rate, Oxygen Saturation, and Blood Pressure.

Background Regularly monitoring common physiological signs, including heart rate, blood pressure, and oxygen saturation, can effectively prevent or detect several potential conditions. In particular, cardiovascular diseases (CVDs) are a worldwide concern. According to the World Health Organization, 31% of all deaths worldwide are from CVDs. Recently, the coronavirus disease 2019 pandemic has increased the interest in remote monitoring. At present, contact devices are required to extract most of an individual's physiological information, which can be inconvenient for users and may cause discomfort. Methodology However, remote photoplethysmography (rPPG) technology offers a solution for this issue, enabling contactless monitoring of the blood volume pulse signal using a regular camera. Ultimately, it can provide the same physiological information as a contact device. In this paper, we propose an evaluation of Vastmindz's rPPG technology against medical devices in a clinical environment with a variety of subjects in a wide range of age, height, weight, and baseline vital signs. Results This study confirmed the findings that the contactless technology for the estimation of vitals proposed by Vastmindz was able to estimate heart rate, respiratory rate, and oxygen saturation with a mean error of ±3 units as well as ±10 mmHg for systolic and diastolic blood pressure. Conclusions Reported results have shown that Vastmindz's rPPG technology was able to meet the initial hypothesis and is acceptable for users who want to understand their general health and wellness.

Measurement of Vital Signs by Lifelight Software in Comparison to Standard of Care Multisite Development (VISION-MD): Protocol for an Observational Study.

BACKGROUND: Measuring vital signs (VS) is an important aspect of clinical care but is time-consuming and requires multiple pieces of equipment and trained staff. Interest in the contactless measurement of VS has grown since the COVID-19 pandemic, including in nonclinical situations. Lifelight is an app being developed as a medical device for the contactless measurement of VS using remote photoplethysmography (rPPG) via the camera on smart devices. The VISION-D (Measurement of Vital Signs by Lifelight Software in Comparison to the Standard of Care-Development) and VISION-V (Validation) studies demonstrated the accuracy of Lifelight compared with standard-of-care measurement of blood pressure, pulse rate, and respiratory rate, supporting the certification of Lifelight as a class I Conformité Européenne (CE) medical device. OBJECTIVE: To support further development of the Lifelight app, the observational VISION Multisite Development (VISION-MD) study is collecting high-quality data from a broad range of patients, including those with VS measurements outside the normal healthy range and patients who are critically ill. METHODS: The study is recruiting adults (aged ≥16 years) who are inpatients (some critically ill), outpatients, and healthy volunteers, aiming to cover a broad range of normal and clinically relevant VS values; there are no exclusion criteria. High-resolution 60-second videos of the face are recorded by the Lifelight app while simultaneously measuring VS using standard-of-care methods (automated sphygmomanometer for blood pressure; finger clip sensor for pulse rate and oxygen saturation; manual counting of respiratory rate). Feedback from patients and nurses who use Lifelight is collected via a questionnaire. Data to estimate the cost-effectiveness of Lifelight compared with standard-of-care VS measurement are also being collected. A new method for rPPG signal processing is currently being developed, based on the identification of small areas of high-quality signals in each individual. Anticipated recruitment is 1950 participants, with the expectation that data from approximately 1700 will be used for software development. Data from 250 participants will be retained to test the performance of Lifelight against predefined performance targets. RESULTS: Recruitment began in May 2021 but was hindered by the restrictions instigated during the COVID-19 pandemic. The development of data processing methodology is in progress. The data for analysis will become available from September 2022, and the algorithms will be refined continuously to improve clinical accuracy. The performance of Lifelight compared with that of the standard-of-care measurement of VS will then be tested. Recruitment will resume if further data are required. The analyses are expected to be completed in early 2023. CONCLUSIONS: This study will support the refinement of data collection and processing toward the development of a robust app that is suitable for routine clinical use. TRIAL REGISTRATION: ClinicalTrials.gov NCT04763746; https://clinicaltrials.gov/ct2/show/NCT04763746. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/41533.

Measurement of Blood Oxygen based on Remote-Photoplethysmography

SpO2, also known as blood oxygen saturation, is a vital physiological indicator in clinical care. Since the outbreak of COVID-19, silent hypoxia has been one of the most serious symptoms. This symptom makes the patient's SpO2 drop to an extremely low level without discomfort and causes medical care delay for many patients. Therefore, regularly checking our SpO2 has become a very important matter. Recent work has been looking for convenient and contact-free ways to measure SpO2 with cameras. However, most previous studies were not robust enough and didn't evaluate their algorithms on the data with a wide SpO2 range. In this paper, we proposed a novel non-contact method to measure SpO2 by using the weighted K-nearest neighbors (KNN) algorithm. Five features extracted from the RGB traces, POS, and CHROM signals were used in the KNN model. Two datasets using different ways to lower the SpO2 were constructed for evaluating the performance. The first one was collected through the breath-holding experiment, which induces more motion noise and confuses the actual blood oxygen features. The second dataset was collected at Song Syue Lodge, which locates at an elevation of 3150 meters and has lower oxygen concentration in the atmosphere making the SpO2 drop between the range of 80% to 90% without the need of holding breath. The proposed method outperforms the benchmark algorithms on the leave-one-subject-out and cross-dataset validation. © 2022 IEEE.

Augmentation of rPPG Benchmark Datasets: Learning to Remove and Embed rPPG Signals via Double Cycle Consistent Learning from Unpaired Facial Videos

Remote estimation of human physiological condition has attracted urgent attention during the pandemic of COVID-19. In this paper, we focus on the estimation of remote photoplethysmography (rPPG) from facial videos and address the deficiency issues of large-scale benchmarking datasets. We propose an end-to-end RErPPG-Net, including a Removal-Net and an Embedding-Net, to augment existing rPPG benchmark datasets. In the proposed augmentation scenario, the Removal-Net will first erase any inherent rPPG signals in the input video and then the Embedding-Net will embed another PPG signal into the video to generate an augmented video carrying the specified PPG signal. To train the model from unpaired videos, we propose a novel double-cycle consistent constraint to enforce the RErPPG-Net to learn to robustly and accurately remove and embed the delicate rPPG signals. The new benchmark “Aug-rPPG dataset” is augmented from UBFC-rPPG and PURE datasets and includes 5776 videos from 42 subjects with 76 different rPPG signals. Our experimental results show that existing rPPG estimators indeed benefit from the augmented dataset and achieve significant improvement when fine-tuned on the new benchmark. The code and dataset are available at https://github.com/nthumplab/RErPPGNet. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Contactless blood oxygen estimation from face videos: A multi-model fusion method based on deep learning.

Blood Oxygen ( SpO 2 ), a key indicator of respiratory function, has received increasing attention during the COVID-19 pandemic. Clinical results show that patients with COVID-19 likely have distinct lower SpO 2 before the onset of significant symptoms. Aiming at the shortcomings of current methods for monitoring SpO 2 by face videos, this paper proposes a novel multi-model fusion method based on deep learning for SpO 2 estimation. The method includes the feature extraction network named Residuals and Coordinate Attention (RCA) and the multi-model fusion SpO 2 estimation module. The RCA network uses the residual block cascade and coordinate attention mechanism to focus on the correlation between feature channels and the location information of feature space. The multi-model fusion module includes the Color Channel Model (CCM) and the Network-Based Model(NBM). To fully use the color feature information in face videos, an image generator is constructed in the CCM to calculate SpO 2 by reconstructing the red and blue channel signals. Besides, to reduce the disturbance of other physiological signals, a novel two-part loss function is designed in the NBM. Given the complementarity of the features and models that CCM and NBM focus on, a Multi-Model Fusion Model(MMFM) is constructed. The experimental results on the PURE and VIPL-HR datasets show that three models meet the clinical requirement(the mean absolute error ⩽ 2%) and demonstrate that the multi-model fusion can fully exploit the SpO 2 features of face videos and improve the SpO 2 estimation performance. Our research achievements will facilitate applications in remote medicine and home health.

Measurement of Vital Signs Using Lifelight Remote Photoplethysmography: Results of the VISION-D and VISION-V Observational Studies.

BACKGROUND: The detection of early changes in vital signs (VSs) enables timely intervention; however, the measurement of VSs requires hands-on technical expertise and is often time-consuming. The contactless measurement of VSs is beneficial to prevent infection, such as during the COVID-19 pandemic. Lifelight is a novel software being developed to measure VSs by remote photoplethysmography based on video captures of the face via the integral camera on mobile phones and tablets. We report two early studies in the development of Lifelight. OBJECTIVE: The objective of the Vital Sign Comparison Between Lifelight and Standard of Care: Development (VISION-D) study (NCT04763746) was to measure respiratory rate (RR), pulse rate (PR), and blood pressure (BP) simultaneously by using the current standard of care manual methods and the Lifelight software to iteratively refine the software algorithms. The objective of the Vital Sign Comparison Between Lifelight and Standard of Care: Validation (VISION-V) study (NCT03998098) was to validate the use of Lifelight software to accurately measure VSs. METHODS: BP, PR, and RR were measured simultaneously using Lifelight, a sphygmomanometer (BP and PR), and the manual counting of RR. Accuracy performance targets for each VS were defined from a systematic literature review of the performance of state-of-the-art VSs technologies. RESULTS: The VISION-D data set (17,233 measurements from 8585 participants) met the accuracy targets for RR (mean error 0.3, SD 3.6 vs target mean error 2.3, SD 5.0; n=7462), PR (mean error 0.3, SD 4.0 vs mean error 2.2, SD 9.2; n=10,214), and diastolic BP (mean error -0.4, SD 8.5 vs mean error 5.5, SD 8.9; n=8951); for systolic BP, the mean error target was met but not the SD (mean error 3.5, SD 16.8 vs mean error 6.7, SD 15.3; n=9233). Fitzpatrick skin type did not affect accuracy. The VISION-V data set (679 measurements from 127 participants) met all the standards: mean error -0.1, SD 3.4 for RR; mean error 1.4, SD 3.8 for PR; mean error 2.8, SD 14.5 for systolic BP; and mean error -0.3, SD 7.0 for diastolic BP. CONCLUSIONS: At this early stage in development, Lifelight demonstrates sufficient accuracy in the measurement of VSs to support certification for a Level 1 Conformité Européenne mark. As the use of Lifelight does not require specific training or equipment, the software is potentially useful for the contactless measurement of VSs by nonclinical staff in residential and home care settings. Work is continuing to enhance data collection and processing to achieve the robustness and accuracy required for routine clinical use. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/14326.

Photoplethysmographic imaging of hemodynamics and two-dimensional oximetry

Background and Objectives: A review of recent papers devoted to actively developing methods of photoplethysmographic imaging (PPGI) of blood volume pulsations in vessels and non-contact two-dimensional oximetry on the surface of the human body is carried out. Results: The physical fundamentals and technical aspects of PPGI and oximetry have been considered. The diversity of physiological parameters available for analysis by PPGI has been shown. The prospects of PPGI technology have been discussed. The possibilities of non-contact determination of blood oxygen saturation SpO2 (saturation of pulse O2) have been described. The relevance of remote determination of the level of oxygenation due to the spread of a new coronavirus infection SARS-CoV-2 (COVID-19) has been emphasized. Most of the works under consideration cover the period of 2010–2021 years. © 2022 Editorial Office of Acta Horticulturae Sinica. All rights reserved.

Evaluation of Remote Photoplethysmography Measurement Conditions toward Telemedicine Applications.

Camera-based remote photoplethysmography (rPPG) is a low-cost and casual non-contact heart rate measurement method suitable for telemedicine. Several factors affect the accuracy of measuring the heart rate and heart rate variability (HRV) using rPPG despite HRV being an important indicator for healthcare monitoring. This study aimed to investigate the appropriate setup for precise HRV measurements using rPPG while considering the effects of possible factors including illumination, direction of the light, frame rate of the camera, and body motion. In the lighting conditions experiment, the smallest mean absolute R-R interval (RRI) error was obtained when light greater than 500 lux was cast from the front (among the following conditions-illuminance: 100, 300, 500, and 700 lux; directions: front, top, and front and top). In addition, the RRI and HRV were measured with sufficient accuracy at frame rates above 30 fps. The accuracy of the HRV measurement was greatly reduced when the body motion was not constrained; thus, it is necessary to limit the body motion, especially the head motion, in an actual telemedicine situation. The results of this study can act as guidelines for setting up the shooting environment and camera settings for rPPG use in telemedicine.

AND-rPPG: A novel denoising-rPPG network for improving remote heart rate estimation.

Heart rate (HR) estimation is an essential physiological parameter in the field of biomedical imaging. Remote Photoplethysmography (r-PPG) is a pathbreaking development in this field wherein the PPG signal is extracted from non-contact face videos. In the COVID-19 pandemic, rPPG plays a vital role for doctors and patients to perform telehealthcare. Existing rPPG methods provide incorrect HR estimation when face video contains facial deformations induced by facial expression. These methods process the entire face and utilize the same knowledge to mitigate different noises. It limits the performance of these methods because different facial expressions induce different noise characteristics depending on the facial region. Another limitation is that these methods neglect the facial expression for denoising even though it is the prominent noise source in temporal signals. These issues are mitigated in this paper by proposing a novel HR estimation method AND-rPPG, that is, A Novel Denoising-rPPG. We initiate the utilization of Action Units (AUs) for denoising temporal signals. Our denoising network models the temporal signals better than sequential architectures and mitigate the AUs-based (or face expression-based) noises effectively. The experiments performed on publicly available datasets reveal that our proposed method outperforms state-of-the-art HR estimation methods, and our denoising model can be easily integrated with existing methods to improve their HR estimation.

Development of a Novel Web Camera-Based Contact-Free Major Depressive Disorder Screening System Using Autonomic Nervous Responses Induced by a Mental Task and Its Clinical Application.

BACKGROUND: To increase the consultation rate of potential major depressive disorder (MDD) patients, we developed a contact-type fingertip photoplethysmography-based MDD screening system. With the outbreak of SARS-CoV-2, we developed an alternative to contact-type fingertip photoplethysmography: a novel web camera-based contact-free MDD screening system (WCF-MSS) for non-contact measurement of autonomic transient responses induced by a mental task. METHODS: The WCF-MSS measures time-series interbeat intervals (IBI) by monitoring color tone changes in the facial region of interest induced by arterial pulsation using a web camera (1920 × 1080 pixels, 30 frames/s). Artifacts caused by body movements and head shakes are reduced. The WCF-MSS evaluates autonomic nervous activation from time-series IBI by calculating LF (0.04-0.15 Hz) components of heart rate variability (HRV) corresponding to sympathetic and parasympathetic nervous activity and HF (0.15-0.4 Hz) components equivalent to parasympathetic activities. The clinical test procedure comprises a pre-rest period (Pre-R; 140 s), mental task period (MT; 100 s), and post-rest period (Post-R; 120 s). The WCF-MSS uses logistic regression analysis to discriminate MDD patients from healthy volunteers via an optimal combination of four explanatory variables determined by a minimum redundancy maximum relevance algorithm: HF during MT (HF MT ), the percentage change of LF from pre-rest to MT (%ΔLF(Pre-R⇒MT) ), the percentage change of HF from pre-rest to MT (%ΔHF(Pre-R⇒MT) ), and the percentage change of HF from MT to post-rest (%ΔHF(MT⇒Post-R) ). To clinically test the WCF-MSS, 26 MDD patients (16 males and 10 females, 20-58 years) were recruited from BESLI Clinic in Tokyo, and 27 healthy volunteers (15 males and 12 females, 18-60 years) were recruited from Tokyo Metropolitan University and RICOH Company, Ltd. Electrocardiography was used to calculate HRV variables as references. RESULT: The WCF-MSS achieved 73% sensitivity and 85% specificity on 5-fold cross-validation. IBI correlated significantly with IBI from reference electrocardiography (r = 0.97, p < 0.0001). Logit scores and subjective self-rating depression scale scores correlated significantly (r = 0.43, p < 0.05). CONCLUSION: The WCF-MSS seems a promising contact-free MDD screening apparatus. This method enables web camera built-in smartphones to be used as MDD screening systems.